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Abstract This work discusses the professional 
outcomes of University of Padova graduates in academic 
year 2007/08. Its aim was to carry out an overall assessment 
of higher education professional outcomes, involving all the 
main actors of the learning process, students, teachers and 
tutors, in a perspective of participatory evaluation, to 
improve teaching and tutoring. The research design 
complements qualitative and quantitative approaches to data 
analysis. First, professional outcomes were analyzed 
according to longitudinal survey data, on a random sample 
of 210 graduates in Educational Sciences, interviewed 
within three years of obtaining their degree. The results of 
two parallel studies, sequential to the quantitative survey, 
are presented, involving 98 professors and 10 tutors in 
evaluation of professional outcomes. In particular, discourse 
analysis was applied to actors' free answers during 
semi-structured interviews, which focused on positive 
aspects and criticism of the courses analyzed, and potential 
improvements to them. 

Keywords University Assessment, Professional 
Outcomes, Quali-Quant Approach, Participatory Evaluation 


1. Introduction 

Before the current educational system was introduced in 
Italy, the implementation of European Union regulations as 
set out in the Sorbonne and Bologna Declarations, in which 
changes were made to teaching autonomy at university level 
(Italian Ministerial Decrees 509/1999 and 270/2004), was a 
great challenge for Italian universities, marked as they were 
by continual changes and adjustments to organizational rules 
and curricula. The so called “Bologna Process”, on the basis 
of voluntary cooperation, agrees and implements common 
objectives for the higher education systems of 47 countries; 
it has transformed the face of European higher education: 
higher education structures have been modified, quality 
assurance systems developed, mechanisms to facilitate 
mobility established, and a range of issues for the social 


dimension of higher education identified. The Bologna 
Process has induced change at systems level through the 
implementation of trust building tools aimed at increasing 
transparency across national jurisdictions and at bringing 
about convergence of systems. These instruments include: 
the three-cycle system and the ensuing development of an 
overarching qualifications framework, the European Credit 
Transfer System and quality assurance (EACEA, 2012). 

Within this process, the Italian universities have 
undergone profound changes, ranging from the passage from 
universities of knowledge to universities of competences 
(Galliani, 2011). In order to monitor training itineraries and 
evaluate the results of the reform (in particular employability 
and the quality of employment), the University of Padova, 
one of the largest in North-East Italy, carried out an 
important and extensive survey to examine the job results of 
graduates who had obtained their first or second-cycle 
degrees in academic year 2007/08. The Faculty of 
Educational Sciences (ES), which played an active role in the 
survey, took this opportunity to benefit by the survey results. 
The faculty research team, composed of scholars from 
various disciplines who are accustomed to using several 
methodological approaches, had a common research aim, 
focused to carry out an overall assessment of learning 
programs, putting particular attention to graduates 
professional outcomes, strongly oriented towards 
interpreting results and consequences in the organization of 
teaching and tutoring. 

The quantitative results of the survey about the learners' 
careers, addressed to graduate students (phase 1 of the 
research design), created the necessary conditions for a 
qualitative study (phase 2), oriented to identify the strong 
and weak points of the learning programs (see Figure 1). 
This second phase involved various actors (teachers and 
tutors) of the educational process in a participatory 
assessment research. 


2. Theoretical Framework 

Nowadays both European and international interest 
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increasingly focused on quality assurance, a system for 
transparency and credibility in higher education for citizens, 
employers, students and researchers from other countries 
(Squarzoni& Stefani, 2011). 

The model of the Bologna Process concentrated on the 
centrality of the person learning: this focus must remain, 
particularly at the moment of evaluation. This crucial 
position of the person who is learning turned out to be one of 
the most innovative and most complex aspects to achieve, in 
the passage between legislation and achievement in daily 
practice (Serbati&Zaggia, 2012). In addition, quality 
evaluation of university teaching takes on value if the 
“product” of the university is understood as a “public good”, 
and if the university system is considered to be a service of 
public utility (Zaggia, 2008), i.e., aiming at improving 
overall wellbeing (Carpitaet al., 2006). 

This research forms part of studies evaluating the quality 
of higher education: specifically, focusing on its efficacy 
(Fabbris, 2007, 2011; Cammelli&Gasperoni, 2012) and on 
the link between education and graduates'jobs (Iezzi, 2011; 
Civardi&Crippa, 2012; Iezzi&Mastrangelo, 2012). The 
theoretical framework contains several interrelated issues, 
and is grounded on national and international literature 
(Biggs, 2003; Frost, 2006; Gruschkaet a/.2006; Semeraro, 
2006; Fabbri, 2007; Masia&Morcellini, 2009; Galliani, 
2011). Although the main focus is on higher educational 
assessment, in particular, the professional outcomes of such 
education, methodological aspects are also examined 
carefully, according to participatory research (Battistellaet 
al., 2004; Bergold&Thomas, 2012; Gubrium& Flolstein, 
1997; Schon, 1987) and the integration of qualitative and 
quantitative approaches (Onwuegbuzie& Leech, 2005; 
Teddlie&Tashakkori, 2009). 

As already noted, professional outcomes are very 
important within an evaluation process. The European 
Community's view of the “double track” i.e., teaching 
system and job market, reveals the need to analyse the 
requirements for professional figures, to define the tools and 
strategies to develop specific professional skills, andthe 
criteria and conditions required to combine job demand and 
job offers (Fabbris, 2008). Graduates' entry into the job 
market and the work they are offered vary according to the 
degree course followed, but job potential is clearly 
connected to the extent of consistency between training aims 
and job market requirements (Palombaet al., 2003; Cedefop, 
2001). Another important point is that the need to 
examinehow graduates can best be inserted into the 
professional job market has recently come into conflict with 
the ongoing crisis in the economic sector, which has severely 
limited further research in this field (Censis, 2012; Cnel, 
2012; AlmaLaurea, 2012). This topic must find its own place 
if a complex evaluative approach is to be carried out. 

In the fields of social sciences and economics, attention 
has mainly focused on the measurement of job outcomes 
according to a “control perspective” (Broadfoot, 2007) 
applied to assessment.However, for an integrated view of 
aspects to be examined when evaluating the professional 


outcomes of higher education, one interesting approach is 
certainly that of participative evaluation! Earl, 2003; 
Gallianiet al., 2009; Zaggia&Maniero, 2009; Grionet al., 
2006; Grion, 2011; Palumbo, 2004; Rossi, 2004, Stake, 1988; 
Wates, 1999). In this approach, the actors of the training 
process themselves play active roles in the evaluative action, 
and thus become its subjects, in their quality and privileged 
position as the first to be the experts on the needs and 
opportunities of the context and training process to be 
evaluated. De Ambrogio (2003) believes that evaluation 
should involve both several actors and several institutional 
levels, linking all the stakeholders involved in training in a 
synergic logic. In the case examined here, students, teachers 
and tutors are the actors of the evaluation: students as the 
prime interpreters of their training, teachers as evaluators 
reflecting on their teaching to their students, and tutors as 
privileged observers as mediators between students, teachers 
and the university system. 

According to the “new” forms of evaluation as defined by 
Varisco (2004: 71), participation consists of understanding 
the processes giving rise to the results, not only on the part of 
teachers, but also by students themselves, in order to create 
properly collaborative "learning environments” in which all 
actors contribute towards improving the processes, products 
and results, from the viewpoint of the improvement 
furnished by their co-participation in the evaluative act. 

The literature provides three approaches to evaluation: 
positivist experimental, pragmatist, and constructivist 
(Zaggia, 2008). As regards the last, Varisco (2002: 158) 
states: “[...] in a constructivist environments, students can 
act in a space using tools, collecting and interpreting 
information, and interacting with the other actors in the 
system (peers or teachers). Learners should be able to define 
aims and learning activities. [...] In a situational 
environment, students can also participate in the practices of 
a learning community. ” (authors' translation). 

This approach also shows its cognitive character, which 
allows the context to be understood by means of the actors' 
own interpretations. A university, as a complex organisation, 
is considered mainly according to such an evaluation, in that 
the sense that actions depend on interactions among the 
actors involved and negotiations of significances (Semeraro, 
2006). 

When defining our research design within this theoretical 
framework, the method guiding our operative choices was 
mixed, with moments of triangulation and integration 
between qualitative and quantitative approaches 
(Onwuegbuzie& Leech, 2005; Teddlie&Tashakkori, 2009; 
Clericiet al., 2013). International debate in the last few 
decades (Datta, 1994; House, 1994; Reichardt& Rallis, 1994; 
Tashakkori&Teddlie, 1998, 2003; Mertens, 2003; 
Marslandet al., 2011) has shown the need to integrate “old” 
and “new” theoretical models. More recently, Lincoln 
&Guba (2000) have mentioned possible “emerging 
confluences” (Vannini, 2009). We considered a mixed 
method approach as the most appropriate way to reach our 
research aim. 
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Figure 1 . Research design 

Table 1 . ES Graduate population and sample size by degree level 


Degree 

Level 

Graduates 

(a) 

Graduates 

(b) 

Sample 

Follow-up 

6 months 

Follow-up 

12 months 

Follow-up 
36 months 

Dl 

405 

325 

186 

161 

145 

134 

D2 

58 

56 

50 

46 

41 

37 

SC 

384 

383 

47 

45 

41 

39 

Total 

847 

764 

283 

252 

227 

210 


(a) 2007/08 graduates; (b) 2007/08 graduates, not attending specialization courses lasting at least one year. 


3. Empirical Research 

The research presented here was developed within the 
perspective of quantitative and qualitative methodological 
integration, using the measurements and analytical tools of 
both, with the participation of the actors involved. The 
quantitative results of the study on job outcomes and 
opinions regarding the quality of university training in a 
random sample of ED graduates from a large Italian 
university were used as the basis for the second qualitative 
phase, involving the teachers and tutors of the academic 
programmes in which the students had obtained their degrees 
(see Fig.l). 

3.1. PHASE 1: Graduate Survey 

The longitudinal survey of 2007-08 University of Padova 
graduates was promoted by the academic authorities in order 
to develop evaluation models and tools to improve teaching 
and educational offers. It is an important source of 
information for evaluating the effectiveness of training 
during first- and second-cycle degree courses, with respect to 
graduates' subsequent employment (Fabbris, 2010). 

The survey plan involved three observation moments (at 
six months, one year and three years after graduation) and 
non-proportional random sampling, both to have a 
representative sample and to keep the costs of data collection 
low. Twelve faculties were involved, with a follow-up of 


about 4,000 graduates. The ES faculty took part in the project 
by surveying four first-cycle degree courses (Dl), two 
second-cycle degree courses (D2) and one single-cycle 
degree course (SC); this sub-sample involved 283 ES 
graduates, 210 being followed for three years after 
graduation. As the survey focused on the job results of 
training, graduates who attended any post-lauream course 
during sample selection were excluded a priori. In the 
various waves of the survey, as some of the interviewees 
were following specialization or other courses lasting at least 
one year, they were also excluded from follow-up and 
filtered a posteriori. Table 1 lists the ES number of graduates 
and sample data during the three types of follow-up. 

The graduate samples was chosen by the 
ServizioStudiStatistici from the university archives by 
non-proportional casual selection, in order to ensure that all 
degree courses were statistically represented and at the same 
time to limit survey costs. In the following analysis, the 
survey data are post-weighted, to obtain a realistic 
reconstruction of the referential population of graduates. In 
this way, the survey could monitor graduates' job itineraries 
through repeated observations up to three years after 
graduation. On these occasions, recent graduates were 
allocated structured questioning paths by means of 
Computer Assisted Telephone Interviewing (CATI). The 
paths differed greatly according to graduates'job status at the 
times of the current and former interviews. In particular, 
those with jobs at the time of the interview were asked to 
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define their work and to mark their profession or job on a list 
(Fabbris, 2005). Opinions about their use of knowledge, 
abilities and skills were studied in relation to their jobs. All 
the interviewees then answered several questions 
highlighting their ex post evaluations about their university 
program and the worth of their degrees. 

Starting from a very large amount of information, interest 
in this research focused on three main issues: (i) graduates' 
professional situation, (ii) the job on which they were 
engaged (if any), (Hi) the adequacy of the university path 
with respect to the job they were doing or expecting to do. 
Many of the degree courses are highly profession-oriented, 
so that some of the in-depth analysis regarded the coherence 
between the dimensions of ‘‘‘expected” professionalism and 
those which graduates declared after they had actually 
gained some experience of certain professions. 


Table 2. ES graduates by upper secondary school and degree. Percentage 
profiles by degree level 


Secondary school 

1 st-cyele 
(Dl) 

2nd-cycle 

(D2) 

Single-cyc 
le (SC)) 

Total 

Professional 

14.2 

3.8 

10.8 

11.7 

Technical 

22.2 

11.3 

16.1 

18.3 

Specialising in 
education 

33.0 

39.6 

45.9 

40.1 

High school 

25.8 

44.2 

24.4 

26.4 

Other 

4.5 

- 

2.8 

3.3 

Total % 

100.0 

100.0 

100.0 

100.0 

N 

288 

353 

53 

694 


3.1.1. GRADUATES’ Characteristics 
The university administrative archives supplied 
information about the sample of ES 2007/08 graduates 
(followed for three years after graduation). Graduates who 


stated that they were “out of the market” were excluded, as 
were those who chose to continue their studies and not enter 
the job market. A weights system was used for analysis, to 
compensate for the unequal probabilities of selection 
between sample data and reference population. The results 
are divided according to the kind of degree course: 
first-cycle (Dl), second-cycle (D2) or single-cycle (SC). 

Women constitute 95% of the sample. The large number 
of female students in the educational field is notable, 
especially in the SC, which trains teachers, whereas there are 
a few more male students (10%) following courses to train 
educators for contexts other than primary school. As regards 
graduates' schooling prior to university, the kind of upper 
secondary school to which they had gone and their high 
school grades (normalized in hundredths) were examined. 
Most of the graduates (Tab. 2) had been to high schools 
specializing in education (40%), of which the degree courses 
followed were a natural continuation, particularly those 
aspiring primary school teachers. 

The distribution was remarkably heterogeneous, with 
substantial numbers of graduates from high schools (26%), 
particularly those focusing on sciences (16%) and technical 
education institutes (18%). There were many high school 
students (44%) among the second-cycle graduates, whereas 
in those of the first-cycle, there were many students with 
technical or professional diplomas (36%). There was a 
considerable difference between graduates’ high school 
grades and the degree courses they chose. Overall, scores 
were distributed over the whole range (60-100), with a slight 
left asymmetry around the mean (79/100) and median 
(78/100). The mean grades for SC graduates were 81.0-83.6 
out of 100 (level of confidence: 95%): they had significantly 
higher scores than both D2 graduates (+4.5; p_value=0.038) 
and, to an even greater extent, Dl graduates (+6.5; 
p_value=0.000). 


Table 3. ES graduates’ characteristics by degree level. Descriptive statistics 


Characteristic 

Degreelevel 

N. 

Min 

Max 

Median 

Mean 

SE 

Age atgraduation 

lst-cycle 

288 

21 

53 

24 

26.9 

.46 


2nd-cycle 

51 

24 

49 

25 

27.2 

.81 


Single-cycle 

353 

22 

35 

24 

25.2 

.15 


Total 

693 

21 

53 

24 

26,0 

.22 

Completion Index 

1 st-cycle 

288 

0.3 

4.3 

1.33 

1.47 

.04 


2nd-cycle 

51 

1.0 

2.0 

1.50 

1.48 

.03 


Single-cycle 

353 

0.3 

2.8 

1.00 

0.98 

.02 


Total 

693 

0.3 

4.3 

1,00 

1.22 

.02 

Graduation grade 

lst-cycle 

288 

84 

110 honours 

98 

99.2 

.37 


2nd-cycle 

51 

98 

110 honours 

110 

108.5 

.46 


Single-cycle 

353 

93 

110 honours 

103 

102.7 

.26 


Total 

693 

84 

110 honours 

102 

101.7 

.23 
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Figure 2. CATPCA. Objects, component loadings and centroids of ES graduates’ characteristics 


There were also notable differences between the 
university careers of these groups, as regards grades and the 
average time it took them to complete their degree course 
and graduate (Tab. 3). Their age at graduation should have 
been influenced by the length of the various types of course 
(3 years Dl; 4 years SC; 5 years D2), but the sequence of 
averages (Dl, 26.9 years old; D2, 27.2; SC, 25.2) suggests 
the influence of other elements. In order to understand this 
phenomenon more clearly, the index representing how long 
it took graduates to complete their training was normalized 
with respect to the standard course duration ( completion 
index: 1 if standard duration was respected; higher than 1 if 
there was some delay; less than 1 if the graduate finished 
earlier than the standard duration). The three groups clearly 
show great differences in the time taken to complete their 
training: the SC graduates were very young and completed 
their paths on time; the D2 and D1 graduates completed their 
courses later than expected, although their performance 
varied widely. As regards their final degree grade, means and 
medians are around 102 over 110, but there were very 
significant differences between the three groups (F=77.9; 2, 
689; p_value=0.000): Dl grades were lower (98-99 over 
110). As already noted, these graduates had a far more 
irregular university career and weaker pre-university 
preparation. 


Figure 2 summarizes graduates’ characteristics, obtained 
by means of an optimal scaling procedure (Principal 
Component Analysis for Categorical Data; for details of 
CATPCA, see Meulman&Fleiser, 2011), which can 
extrapolate the main dimensions of a multidimensional 
structure of data of various types, both metric and non-metric. 
The plot shows the object points (cases), component 
loadings for variables with an optimal scaling level which 
results in vector quantifications (high school grade, 
graduation grade, age at graduation, completion index) and 
centroids for the variables with an optimal scaling level 
(course, degree level, secondary school). 

The first two dimensions are highly reliable (Cronbach's 
alpha : 0.74 for the first dimension and 0.65 for the second). 
The case positions (labelled according to degree level), show 
component loadings and centroids on the plane defined by 
those main dimensions, so that the first dimension is defined 
by pre-university features (high school grade and type of 
secondary school): it is representative of the quality of the 
pre-university’ path. The second dimension is characterized 
by the type of degree, its duration (completion index) and 
graduation grade, which indicates the type and quality of the 
university path. 

Classifying the graduates into the three types of degree 
course turned out to be very efficient, in that some aspects of 
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pre-university paths and associated features of efficiency (in 
terms of timing) and quality (through the final graduation 
grade) could be captured. 


Table 4. ES graduates’ professional careers. Percentage profiles by degree 
level 


Professional careers 

D1 

D2 

SC 

Total 

Subjects kept jobs they had before 
graduating 

35.8 

36,5 

19.0 

27.3 

consistent 

19.7 

30.8 

18.9 

20.1 

regularly employed 

32.3 

32.7 

19.0 

25.5 

permanently employed 

28.1 

31.4 

19.0 

20.0 

Subjects found new jobs (different 
from their old ones) 

15.3 

26.9 

45.9 

31.7 

consistent 

10.7 

19.2 

43.2 

27.9 

regularly employed 

11.1 

25.0 

45.9 

29.8 

permanently employed 

8.0 

19.6 

7.5 

5.2 

Subjects found their first jobs 

40.3 

36.5 

35.1 

37.4 

consistent 

29.4 

19.2 

35.0 

31.5 

regularly employed 

32.6 

30.8 

35.1 

33.7 

permanently employed 

12.8 

19.6 

11.5 

13.8 

Unemployed (used to have a job) 

6.9 

- 

- 

2.9 

Unemployed (never had a job) 

1.7 

- 

- 

0.7 

Total % 

100.0 

100.0 

100.0 

100.0 

N 

288 

51 

353 

694 


3.1.2. GRADUATES’ Professional STATUS 

The data collected six months and one year after 
graduation already showed that a high percentage of the 
graduates were already employed, thus confirming the 
results of other surveys (e.g., Istat, 2010; AlmaLaurea, 2011) 
on Italian graduates employability in the educational field. 
The survey estimates that, three years after graduation, only 
3.5% of subjects are not employed, and only about 1% are 
never observed as in work: all these cases involve D1 
graduates. These results are remarkable, considering the 
frequently changing situation of graduates’ employability, 
although both their more or less linear job itineraries and the 
quality of their jobs must also be considered. 

When data about subjects' professional status at 
graduation and 6, 12 and 36 months later (i.e., the post 
lauream points of observation) are combined, the 
professional paths of graduates over the three-year period 
can be re-created (2007/08 to 2010/11). For those in work, 
their job stability and any job changes can be verified. Table 
4 shows both summarized (last column) and comparative 
indexes (specific rates by degree level)about graduates’ 
types of degree and professional status three years after 
graduation. Table 4 also shows subjects'job suitability at the 
moment of the last data collection, with respect to the 
university course followed and the kind of contract currently 
regulating their current jobs. 


The first path shows for how long, after graduating, 
subjects kept the job they had already been doing. More than 
one-third of D1 graduates kept the jobs they had before 
graduating: they were not always consistent with their 
university courses, but were usually “secure”, meaning that 
they were had proper work contracts, even if they were not 
permanent. D2 subjects who stated that they were already 
working when they graduated (36.5%) continued to have 
good jobs, secure, permanent and usually consistent. Of the 
SC graduates, fewer than 20% had jobs and all of them were 
already teaching under permanent contracts. 

About 70% of the graduates had found new jobs (either for 
the first time or by changing their old jobs) within three years 
of graduating. Among the SC graduates, this percentage rose 
to over 80%, with very high levels of consistence, although 
permanent contracts were unusual. Among D1 and D2 
graduates, the percentages were lower (56% and 63% 
respectively) and with lower coherence levels (about 40%). 
However, permanent contracts were more common. 

In general, we note that jobs for D1 graduates were less 
well-defined than the others. There were also a small number 
of cases of unemployment. 

3.1.3. GRADUATES’Jobs 

We now focus on subjects who stated that they had jobs 
three years after graduating. Our first aim was to classify free 
answers to questions concerning jobs, which were usually 
phrased very briefly in imprecise terminology. The faculty 
tutors - i.e. the coaches who help students to learn how they 
can best learn and how to work in an academic environment, 
were involved in processing and interpreting the data. They 
used both structured data and graduates' free answers to 
describe their post lauream condition, and extrapolated 
information which was not always explicit in single answers, 
particularly as regards their jobs and the coherence between 
the teaching aims of degree courses and their professional 
collocation. The perspective of classical content analysis 
(Krippendorff, 1980; lan, 2004; Tuzzi, 2003) was applied, 
starting from an ex ante conceptual grid composed of the 
above-mentioned list of the faculty's typical professions. 
However, as many parts of the grid (and thus the list) turned 
out to be inadequate, we proceeded by examining 
respondents' self-collocation in the professions list as one of 
the elements (not the only nor the main one) to be considered 
when classifying the type of job. We then found a set of 
informative variables (economics, public or private sector, 
final users) which contributed towards tracing job contexts 
and features. This enabled us to reformulate the list of 
graduates’ jobs, shown briefly as macro-categories in Table 
5. 

There were clearly two main professions: for SC graduates, 
teaching (primary school, preschool and special needs 
educational assistance), plus educators in the social and 
health services sector and extra-school sector for D1 and D2 
subjects. The latter included some examples of consultancy 
and managerial work in contexts pertinent with university 
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Items were sent to all subjects and submitted according to 
a self-anchoring scale from 1 (minimum) to 10 (maximum). 
Figure 3 shows the distribution of answers, emphasizing a 
few outliers which express very strong dissatisfaction. Most 
of the graduates were satisfied with their training 
(median=8.5; trimmed mean=7.7), although inter-subjective 
variability was high (range 4-10, standard deviation=1.9, 
interquartile range=2). 


Figure 3. Adequacy of university training. (Boxplot) 

The relation of dependence between the adequacy 
evaluation of training and the various outcomes along 
subjects' professional path was examined by analysis of the 
variance of adequacy evaluation by certain factor variables 
and covariates with the general linear model univariate 
procedure (for details, see IBM, 2011). The factor variables 
divide the population into groups, so that this model can test 
null hypotheses about the effects of those factor variables on 
the means of various groupings of a joint distribution of 
independent variables. This enables us to compare the means 
of respondents’ subgroups, classified by degree course and 
the characteristics of their professional path (type of path, job, 
type of contract) while taking into account the effects of the 
various qualities of the respondents with respect to their 
perfonnance: high school and graduation grades, completion 
index. 

Table 6 shows the full factorial model, containing all 
factor and covariate main effects and factor-by-factor 
interactions. The overall adaptation was acceptable, at 
R 2 =0.49. 


Table 6. Adequacy of university training. ANOVA by GLM: full factorial model with covariates 


Effect 

Variables 

F 

pvalue 

Factor 

Professional path 

4.581 

.000 

main effect 

Job 

6.802 

.000 


Form of contract 

6.308 

.000 


Degree course 

6.084 

.000 

Covariate 

High school grade 

2.327 

.128 

main effect 

Graduation grade 

14.255 

.000 


Completion index 

5.272 

.022 

Interaction 

Professional path * Job 

29.399 

.000 

effect 

Professional path * Form of contract 

3.892 

.002 


Professional path * Degree course 

1.634 

.123 


Job * Degree course 

0.799 

.372 


Fonn of contract * Degree course 

2.766 

.027 


Professional path * Form of contract * Degree course 

1.549 

.201 


i -1-1 i-1-1-1-1-1-r 

1 2 3 4 5 6 7 8 9 10 


paths. 


Table 5. ES graduates’jobs. Percentage profiles by degree level 


Job 

D1 

D2 

sc 

Tot. 

Teacher 

13.0 

12.6 

97.3 

56.0 

Educator 

49.3 

29.4 

- 

22.7 

Coach/Responsible for human 

resources 

4.9 

33.4 

- 

4.5 

Coordinator (educational services) 

3.1 

2.4 

- 

1.5 

Consultant (educational sector) 

2.0 

6.1 

- 

1.3 

Other (partially consistent) 

7.2 

9.1 

- 

3.7 

Other (non-consistent) 

11.8 

7.8 

2.7 

6.9 

Total number of graduates with jobs 

91.5 

100.0 

100.0 

96.5 

Unemployed 

8,5 

- 

- 

3.5 

Total 

100.0 

100.0 

100.0 

100.0 


These professional outcomes (which closely followed 
study paths) showed a remarkable number of “other” 
activities, many of which were mentioned by D1 subjects. 
Their description of these activities showed that working as a 
clerk was an alternative outcome to roles such as “coach”, 
“tutor” or “person responsible for human resources”, and the 
respective tasks and duties had to be examined more 
thoroughly in order to evaluate their consistency level. 
Among these “other” activities, “partially consistenf’were 
distinguished from “non-consistent” ones by a set of answers 
illustrating jobs with respect to the economic sector, public 
or private sector, and the kinds of users of a certain type of 
job. These partially consistent activities could become 
completely pertinent in parallel to the pursuit of a 
professional career. 

3.1.4. Adequacy of University Training 

Our graduates answered one question which explored their 
ideas about the adequacy of the university training they had 
received according to their job (current or anticipated). 
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Kept job occupied before graduating (consistent) 
Found new job (consistent) 
Found first job (consistent) 
Kept job occupied before graduating (not consistent) 
Found new job (partially or not consistent) 
Found first job (partially or not consistent) 
Unemployed (used to have a job) 
Unemployed (never had a job) 

Coordinator (social and educational services) 

Educator 

Person responsible for human resources 
Special needs teaching assistant 
Preschool teacher 
Primary school teacher 
Consultant (educational services) 
Coach 

TraineeJCommunity service 
Other 

Looking for a job 

Permanent job 
Non-permanent job 
Self-employed 
Atypical job 
Apprentice 

OVERALL 



Figure 4. Adequacy of university formation. Expected marginal estimate means (95% confidence intervals) 


All the factor main effects and some of the interactions 
had high levels of significance, showing that there were 
differences among the means. The set of covariates in the 
model allowed us to make standardized comparisons with 
the typical high school grade (79.2), graduation grade (101.8) 
and completion index (1.2). 

Figure 4 shows the 95% confidence intervals of the 
expected marginal estimate means of adequacy evaluation. 
The graduate groups which plot far from the overall mean 
(estimated at 6.9-7.3) are clear-cut, and differences between 
the groups, which must be considered when the intervals do 
not overlap, can easily be checked. 

In respect to the principal issue of interest, analyzed by the 
survey among the graduates, we can summarize these 
principal results: (i) graduates’ professional careers have 
very heterogeneous consistency and regularity, and the 
employability strictly depends on the level and type of 
degree course; (ii) the jobs on which graduates are engaged 
tree years after their graduation are frequently what it is 
expected in relation to their learning program, but there is 
also a remarkable percentage of “other” jobs that are 
partially or completely non consistent with the expected 
professional outcome of the degree program; (Hi) the 


graduates’ evaluation of adequacy of the university path are 
in general positive, but they strictly depend on the specific 
learning program they attended, the job and the form of 
contract that regulates the professional condition, the 
professional path from the degree completion to the current 
job. 

The survey highlights that some degree courses, in 
particular some first-level degrees, not always succeed in 
defining clear professional outcomes. This evidence founds 
the need of an in deep analysis of the learning process and its 
learning outcomes. The actors of the learning process should 
increase their awareness on the results of their teaching and 
training activities, and become part of a process which may 
improve the quality and effectiveness of the system. 

3.2. PHASE 2: The Qualitative Study 

The findings of the first analysis were presented to the 
professors teaching the first-cycle degree courses (Dl). We 
examined courses identified as showing some weak points; 
i.e., low scores for students’ high school qualifications at the 
start and for their degree at the end of their university career, 
and their low or inconsistent job opportunities. We asked 
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their professors to read these findings and to provide written 
answers to four correlated questions: 

1) What do you think about these results, particularly 
those relating to the job opportunities of graduates 
who completed your degree course? 

2) Reading the results of the 2007/08 graduates' job 
opportunities and their comments on their university 
education, which strong points do you identify related 
to the degree course which you teach? 

3) What weaknesses do you identify in that course? 

4) In your opinion, are there any improvements (at 
individual or course level) which could be 
implemented in the new university system? 

This phase specifically aimed at obtaining various types of 
data and integrating them with the results of quantitative 
analysis, for deeper understanding of reality. Professors’ 
answers were collected and submitted, together with the 
quantitative findings, to the faculty tutors. The tutors were 
then to interpret them and deduce from them unvoiced 
information about academic activities and the level of 
consistency between the educational aims of the degree 
courses and the graduates’ subsequent professional 
employment. 

In order to collect the comments made by tutors, they were 
provided with a list of the questions given to the professors 
but were allowed to develop their personal viewpoints and 
highlight any significant elements without limitations 
imposed by our input. It should be stressed that the tutors’ 
involvement in this study was an important educational 
occasion for them. They had the opportunity to reflect on and 
gain understanding of the job outcomes of the various degree 
courses, as a consequence of which they could be better 
prepared to do their own jobs, such as orienting students at 
the start of their university career and assisting them 
throughout it. 

From the research point of view, the tutors’ professional 
status, between that of professors and students, represented a 
privileged perspective in observing the dynamics 
characterising courses and captured their protagonists’ 
“perceived realities”. These realities would be more 
significant because: “[...] how things are is often less 
important than how people think - or perceive - things are 
[...] because it is these perceived “realities ” that give shape 
and meaning to people’s lives and actions” (Flutter 
&Rudduck, 2004: 6). 

The professors’ written answers and the tutors’ comments 
were then submitted to Discourse Analysis (DA) (Flardly et 
al., 2004) by AtlasTi software, which allows systematic 
analysis of complex phenomena hidden in textual data. We 
chose to organize these texts into two Flermeneutic Units 
(FlU) and analyze them separately because of their different 
sources and characteristics: the professors’ brief answers to 
precise questions versus the tutors’ longer, more pondered 
texts. 

Discourse analysis of the texts started with our 
time-consuming reading and rereading of the above 
materials many times (Nikander, 2008). The method 


recommended by Perakyla (2005: 870) was followed: “in 
many cases, qualitative researchers who use written texts as 
their materials do not try to follow any predefined protocol 
in executing their analysis. By reading and rereading their 
empirical materials, they try to pin down their key themes 
and, thereby, to draw a picture of the presuppositions and 
meanings that constitute the cultural world of which the 
textual material is a specimen ”. After this first phase, 
following the recommendations of Cohen et al. (2007) on 
DA, and in order to avoid oversimplification, we used a 
bottom-up process to derive a list of codes 1 , one for each FlU, 
representing the meanings emerging from the texts. This 
process revealed patterns and broad areas in the discourses. 
We then re-examined the texts to discover the intentions of 
interviewees, the functions and consequences of the 
discourses, using tools as “families” and “networks of 
meanings” in order to create an interpretive framework in 
which to understand the social reality created by participants. 

3.2.1. Professors’ Point of View 

All 98 professors teaching at least one of the four degree 
courses considered in the quantitative survey were asked to 
provide written free answers to four quoted questions in an 
online interview developed through the LimeSurvey 2 
application program. After this invitation had been sent out, 
professors who had not answered were contacted again and 
offered the opportunity to provide written free answers 
through a paper interview. The total response rate was 56%: 
the highest response rate was for adjunct professors (76%) 
and the lowest for full professors (40%) and researchers 
(43%); the rate for associate professors was good (64%). 

The 55 respondents commented on 71 courses: 25% 
first-year, 34% second-year and 41% third-year, thus 
denoting the increasing involvement of professors in the later 
years of courses, which are typically those when lectures 
tend to cover more professional aspects. The professors were 
asked to read the data concerning job opportunities for their 
graduates before the implementation of the new system of 
university degrees, and to express their opinions, particularly 
by highlighting any strengths, weaknesses and possible ways 
of improving the new system. The complete list of codes 
developed through the DA of respondents' answers 
(described above) was classified into three ’’families” 3 : a) 
strengths, b) critical aspects, and c) improving actions (Fig. 

5). 


1 Coding simply involved associating code words with selections of 
textual data (quotations). 

2 LimeSurvey is an open-source, free software application which allows 
users to create online surveys. For details, see: 
https://www.limesurvey.org/en/. 

Families are a way of forming clusters of codes for easier handling of 
code groups. 
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Figure 5. Network of meanings emerging from professors’ answers 


The first family contained the representative codes of the 
text extracts referring to and features of degree courses 
which the professors identified as qualifying and positive. 
These codes can be organized according to three focuses: the 
first, which is the most frequent, relates to many positive job 
opportunities, expressed by codes such as “good job 
opportunity”, “opportunity of various job placements” and 
“job opportunities which are consistent with type of degree”; 
the second refers to the positive features of degree courses, 
denoted by codes such as “appropriateness and completeness 
of basic teaching”, “formation of a forma mentis” 
(professional culture acquired at university) and “meaningful 
basic teaching”; the third is associated with “personal 
motivation”, which is the basic reason why many students 
decide to attend degree courses aiming at forming socially 
useful professional figures. This last aim includes the context 
of supportive relationships with young people and adults 
who are going through difficult situations. 

The second family of codes comprises those which 
represent aspects of degree courses considered by professors 
to be weak or negative. Also in this case, the codes can be 
organized into two focuses referring to the causes of negative 
aspects. The first focus groups the codes (hereafter listed 
according to decreasing order of frequency) which highlight 
internal elements of university courses which may lead to 
graduates' poor professional success: “an inappropriate 
curriculum for the requirements of the job market”, “lectures 
are poorly organized and managed”, “formation of a general 
rather than a specific professional figure”, “poor connection 
and association between teaching and internship, between 


theory and practice”, and “poor entrepreneurial competence”. 
The second focus concerns external causes. First of all, in 
both public and private workplaces, professional figures are 
often poorly acknowledged and esteemed, so that graduates 
are often hired to do temporary jobs and are underpaid. 
Secondly, there is a prevailing idea that the training of 
students enrolling at university has worsened over the last 
few years, due to the uncertainty of some basic skills - 
particularly language and communications - weak general 
knowledge, and a poor basic ability to study and conduct 
research. 

The third family is composed of codes which identify the 
text extracts in which professors highlighted potential 
corrections which could improve the general offer of 
teaching and courses. The codes in this third family cover the 
particular attention which professors paid to the need to 
re-organize courses in order to provide more professional 
training, to enhance the relationship between internship and 
teaching, where the university offers paths of knowledge and 
the opportunity to participate in professional environments 
through various forms of cooperation with institutions and 
people external to the university which could represent 
future job opportunities for graduates; and where there is 
teaching and subjects which are closely linked to the job 
market, in order to discuss important issues such as recent 
legislation. 

Lastly, as regards potential corrections, a significant 
proportion of professors mentioned the need for greater 
cooperation among professors teaching different subjects 
and therefore better integration among subjects and teaching. 
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Figure 6. Network of meanings emerging from tutors’ texts 


3.2.2. Tutors’ Points of View 

The opinions of six of the ten tutors were examined by 
qualitative analysis, as described above. The different types 
of text and the greater number of details provided by the 
tutors than the professors enabled us to organize the codes 
according to relations inferred on the basis of the meanings 
“emerging” from the text extracts. These relations were then 
developed inductively from analysis of the plots included in 
the narrations. As in the case of the professors, the list of 
codes developed through DA of the answers was organized 
into three families: a) strengths, b) critical aspects, c) 
improving actions (see Fig. 6). 

Although numbering fewer than the critical aspects, some 
important strengths also emerged from the analysis, 
supporting findings from the quantitative analysis. The tutors 
of the undergraduate courses (Dl) describe the following 
aspects as qualifying: the job satisfaction which former 


students feel in their current job; perception of the social 
utility of their profession; and the match between graduates' 
cultural interests and their forma mentis. Tutors highlighted 
the satisfaction reported by former students with the brief 
time it took them to find a job, with the type of job they found, 
and with the good match between their training and later job 
opportunities. 

The highest number of meanings emerging from the texts 
concerns the critical aspects of degree courses; in other 
words, aspects which require “rethinking” courses in order to 
improve them. The codes “poor competence in professional 
practices” and “theoretical-professional gaps” describe the 
areas of skills and competences which the tutors, like the 
professors previously, view as aspects which their graduates 
lack when facing the job market. After this analysis, the 
tutors mentioned possible corrections which could solve 
these critical aspects. Like the professors, the tutors' view 
was that the best thing was to improve “the dialogue between 
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university and the local environment, with regard to required 
professional aspects” and thus adapt training to the specific 
requirements of the organizations and institutions which 
represent the normal job opportunities for graduates. By 
highlighting the lack of “tuning”, or the “no or low 
consistency between university formation and job market”, 
the tutors believe that “ validation of an operator's skills 
should be less associated with formal procedures and 
increasingly depend on the overall quality of training. 
University courses should be therefore based on the demand 
for services”(authors' translation). 

In this setting, the tutors suggested that courses need to 
“place more value on internship and professional aspects” 
and that graduates must have the opportunity to “experience 
integration, confrontation, mediation and cooperation on 
several levels in various multi-professional environments, 
both before and after internship (in order to develop critical 
thinking about their profession)” and to “organize lectures 
(ex ante, in itinere and post courses) which ask professional 
figures from the specific workplace of a training team, so 
that both graduates and professors can develop, achieve , and 
analyze training projects concerning the specific contexts to 
which the professionals belong”. According to the tutors' 
opinions, a greater use of tools and contexts typical of 
professional figures, e.g., educational and heuristic tools, can 
improve the definition of a professional figure and, above all, 
lead to graduates' greater awareness of the training they have 
received, particularly the tasks and limitations of their 
professional profiles. 

3.2.3. Some Reflections on Qualitative Analysis 

Findings from the qualitative analysis gave rise to several 
important comments. The most significant criterion used by 
the professors in evaluating the strengths of degree courses 
(the first family of codes) regards those features of the 
courses which allow students to become acquainted with the 
competences which are now required by the job market. In 
other words, most of the professors consider degree courses 
which prepare students to face real employment contexts are 
effective, so that students are offered more job opportunities. 
Therefore, courses which are positively evaluated are those 
whose graduate students found employment consistent with 
their university studies. These courses are clearly highlighted 
in the quantitative analysis, which describes both the 
coherence and adequacy of university training. Instead, 
according to the professors, courses which need to be 
“re-organized” are those whose graduates are not doing jobs 
consistent with their university program or who are not 
improving their previous professional status (for students 
who worked before graduating). The professors' main 
criticisms concerned the low level of salaries and, in 
particular, the precarious employment of graduates. 
Flowever, it should be noted here that this situation, in Italy, 
does not only involve educational professions and is in fact 
widespread. 

We also stress that, within the context of the Italian 
academic tradition, the actual professional potential of 


courses (at least as regards the teaching professions) have 
attracted increasing attention since the university reform 
came into force, introducing three-year courses (i.e., the 
first-cycle degree course). Therefore, our professors showed 
full awareness of this situation and have sound knowledge 
which enables them to interpret the required improvements. 
Conversely, their positions change when they are asked to 
identify strengths and weaknesses, which are more closely 
related to the educational features of courses. The crucial 
problem is the need to reach equilibrium between good 
general knowledge (related to acquisition of cross-curricular 
competences) and adequate vocational training; the latter 
aspect is deeply rooted in real traineeship experiences in job 
environments, and may be integrated with appropriate 
theoretical reflections in academic settings. 

As regards the second family of codes, of great interest is 
the fact that much responsibility for criticism concerns the 
design, organization and management of degree courses. In 
particular, very many professors noted weak or non-existent 
coordination among the various educational units (there are 
several teaching units each year), because of the lack of 
coordination among teachers whose subjects do not share the 
same contents and methods. In line with several 
considerations developed in relation to the first family of 
codes, some professors expressed negative opinions about 
the ways in which theory (or general pedagogical knowledge) 
is integrated with practice (or professional competence) in 
degree courses. The tutors also support this idea and focus 
their arguments on the critical area of the relationship 
between theory and practice within the framework of 
education for, and in, the professions (Fabbri, 2007). Some 
professors even expressed doubts about the possibility of 
universities offering good vocational training. They believe 
that this can only be acquired through direct experience in 
authentic job realities. 

The third family of professors’ answers confirms previous 
comments on the relationship between theory and practice: 
they express the need to improve links between 
theoretical/pedagogical and professional knowledge. 
Nevertheless, their suggestions are not limited to the simple 
hope of improving and strengthening the current features of 
teaching courses. Great attention focuses on traineeship 
(currently mandatory for all students, for a minimum of 250 
hours): almost all subjects believe that educational potentials 
can be improved by paying greater attention to the phases of 
planning and evaluating courses and by involving tutors and 
mentors in more training activities; the latter represent 
particularly significant actors for effective career guidance 
and the productive insertion of graduates into job 
environments. In order for graduates to put into practice their 
acquired theoretical knowledge to good effect, many of the 
professors stated that improved training is not enough: there 
is a need for deeper collaboration between universities and 
the companies which accept graduates as apprentices. 

To summarize, many answers reveal clear awareness of 
the fact that universities must interact and develop a better 
dialogue with the external environment. This awareness 
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involves several realities, from the job market to the 
economic and political system and public administration. In 
particular, it represents a series of new tasks for Italian 
university professors, which goes beyond the traditional 
aspects of teaching and research. Awareness that changes in 
university life - aspects which have often occurred in a 
confused and contradictory way - are irreversible, is often 
associated with anxiety and concern. Some professors 
wished that a new organization (or revision) of degree 
courses would take as its starting point competences required 
from professional profiles; others expressed considerable 
resistance to the idea of a university curriculum which is 
functional to the job market. According to a centuries-old 
tradition, those last mentioned believe that university 
training should aim at developing second-level competences 
(cultural, reflective, meta-reflective) and therefore reject 
subordination of these skills to the present-day economic 
centered world. 

According to the tutors, it is important to rethink the 
training of professional figures according to new 
professional types of professional epistemology, following 
Schon (1987), according to whom practice is acknowledged: 
as an epistemological and historical context in which 
knowledge is transferred and developed: a social and 
physical space in which learning and working take place; an 
analytical construct which contributes to the explanation of 
how one learns and develops professional knowledge (Fabbri, 
2007). By rejecting the technical rationale according to 
which professional figures are instrumental problem-solvers 
who can find the best tools for specific aims, these new types 
of epistemology question the view of training seen as a 
model of abstract and explicit knowledge transfer, from the 
mind of a person who already knows, to the mind of a person 
who does not yet know, in environments which do not 
include the complexity of practice and the existence of 
knowledge in communities of practitioners. In this way, as 
stated by Fabbri (2007: 62-63): “Training loses all its 
decontextualized features and becomes situated training; a 
device which is neither top-down (applicative paradigm) nor 
bottom-to-bottom, but is concerned with developing 
communities of practice towards reflective competences 
which allow the acquisition of awareness of [...] 
professional practices” (authors' translation). 

4. Considerations on Method and 
Findings 

This study on graduate students carried out by the research 
group of the Faculty of Educational Sciences is an 
opportunity to promote actions aimed at improving the 
context examined. The main aim of this study was in fact to 
make a critical and pro-active contribution. 

The main improvement took place when quantitative 
analysis of the data was integrated with qualitative analysis. 
The former provided an interesting picture of graduates' 
characteristics, taking into account their training and the 


variety of pathways into employment. This enables us to 
highlight some of the aspects of the quality of education, in 
particular-in this paper- the match between university 
curriculum and subsequent job, and between training 
received and work performed. This knowledge about the 
number of graduates with different characteristics allows us 
to identify, for example, the characteristics of teacher 
education. This is a clearly job-oriented course(regarding the 
possibility of entering the job market and the quality of 
training), although it raises some external problems, such as 
difficulties in finding legal forms of work contracts. The 
quantitative analysis then showed that these strengths are not 
evenly shared by other training courses in both first- and 
second-cycle degree courses. 

However, we believe that a purely quantitative approach 
cannot be effective when we endeavor to develop reflective 
and self-evaluating processes in the actors in the context 
examined. A quantitative survey is not suitable for collecting 
comments and suggestions from teachers and tutors: first, 
because they are much smaller populations, with different 
abilities to respond(our 55 professor respondents are not 
representative of the 98teachers involved in the courses) and, 
secondly, because the research aim is different: we did not 
wish to quantify the opinions of these actors, but to involve 
them in reflection and proposals. Thanks to knowledge based 
on quantitative research, this reflection starts from the actual 
picture, graduates' true stories, the true dimensions of their 
career paths. 

The motivation to integrate qualitative and quantitative 
research often stems from a wish to construct(by means of 
typically qualitative methods) paradigms of reference on 
which to structure subsequent quantitative research. That is, 
the former assumes a service role for the latter. At other 
times, a qualitative approach is preferred, and quantitative 
analysis only helps to present the context. In the present 
work, researchers from various disciplines, who habitually 
use various methodological approaches, met around a 
common research goal, strongly oriented towards 
interpreting results and consequences in the organization of 
teaching. This is a research process, in which two different 
perspectives in interpreting the same reality can contribute 
significantly to further knowledge of that reality. According 
the research and its results, we can now list some important 
implications for future research perspectives. 

First, we believe that sharing results, both quantitative and 
qualitative - that is, the new knowledge generated by this 
study - with the members of the contexts which were 
examined, represents an important moment in the paths 
involved in the research. In particular, in participative 
research, researchers collaborate with stakeholders in order 
to generate better understanding of situations to improve 
actions aimed at enhancing practice. In line with this idea, we 
believe that sharing final results is important for the 
development of new reflective paths for professors and tutors, 
a necessary condition in order to make definite 
improvements to course offers. 

Second, the aim of gathering further knowledge on the 
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subject by analysis of qualitative data may be developed 
further by extending the qualitative survey to graduate 
courses not considered in our research. This may develop 
other reflective improvements in different contexts. 
Extension to a larger sample of subjects-students, teachers 
and tutors-and collection of their free expressions on the 
problems which emerged as significant in exploratory 
analyses (sections paragraphs 3and 4)could offer the 
possibility of mixed analysis (quali-quant) of data collected 
in unstructured form, typically associated with the 
qualitative approach. In this way identification of emerging 
conceptual categories could be accompanied by a measure 
of the prevalence of topics and variability between groups 
regarding their importance. Relationships between concepts 
could be analyzed with the classical tools of analysis of 
multiple correspondences, and the causal links assumed in 
the diagrams presented here could be measured by their 
strength and direction. But in order to do this, as well as a 
representative sample of respondents, texts must refer to the 
same units of analysis used by all researchers. Fragments of 
the text corpus can no longer be the unit of analysis. The 
individual(or interview) should be the common reference in 
the analysis and presentation of data. In this way, the 
information will become consistent and fully integrated. 

At a methodological level, our main aim was to overcome 
these two partial approaches by combining them with a third 
dimension, known as organizational improvement, which 
focuses attention on actors’ potentialities and their 
environment in order to develop improvement strategies. In 
this way, it is possible: (i) to define an integrated model 
which can report and provide the information and data 
required for actors to express founded opinions on higher 
education (aims, activities, learning environments, results 
and management systems); (ii) to induce stakeholders to 
develop reflective processes aimed at determining (and 
creating the related awareness) of the strengths and 
weaknesses of graduate courses, in order to promote changes 
and improvement strategies. 

After implementing the appropriate refinements, 
particularly with reference to critical aspects related to 
methodology, previously described, we believe that this 
research is an interesting example of self-assessment of 
university courses. One of its most important aspects is that it 
overcomes some of the main weaknesses of the assessment 
process which universities have adopted and which focuses 
mainly on external assessments. It also aimed at 
recommending improvements, following an approach based 
on participation and mutual sharing of choices and possible 
solutions. 
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